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LOW BANDWIDTH TELEVISION 

CROSS-REFERENCE TO RELATED APPLICATION 
This application claims the benefit of U.S. Provisional Application No. 
60/071,930, filed on January 20, 1998 and entitled "LOW BANDWIDTH 
TELEVISION". 

BACKGROUND OF THE INVENTION 

1. Field of the Invention 

The present invention relates generally to image processing, and 
specifically to production of images and audio in a personal computer 
environment. 

2. Discussion of the prior art 

An important issue in digital technology is providing video images on a 

personal computer. These images are transmitted across the Internet and 

other networks, across telephone lines with modem-to-to modem connections, 

or received from compact disk read-only memories (CD-ROMs). The speed of a 

modem is commonly the limiting factor in sending real time, continuous video 

information across the Internet, over corporate intranets or local area 
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networks. In comparison, continuous network transmission of audio data does 
not present significant difficulties. 

Table 1 shows theoretical bandwidth maxima for various network 
architectures. Modem-to-modem connections across lines in plain old 
telephone service (POTS) have a theoretical bandwidth of 3,360 bytes per 
second, while connections across the Internet with a modem or single ISDN are 
limited to 5,600 bytes per second. Dual ISDN network architectures transmit a 
maximum of 1 1,200 bytes per second, while corporate local area networks with 
lOBaseT connections have a capability of transmitting one megabyte per 
second. With the exception of telephone line connections, these other 
techniques involve non-continuous, packet- switched data. Satellite and cable 
architectures are also possible, but have not yet been widely adopted and 
present other difficulties. 



Network Architecture 


Theoretical Maximum 
Bandwidth 


Continuous Bandwidth 


Modem to modem over POTS 


3,3360 bytes/sec 


Yes 


Internet with modem 


5,600 bytes/sec 


No 


Internet with single ISDN 


5,600 bytes/sec 


No 


Internet with dual ISDN 


11,200 bytes/sec 


No 


Corporate lOBaseT 


One megabyte/ sec 


No 
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Table 1 

On the other hand, computer memories and processor speeds have made 
rapid advances. Personal computers have hard drives accommodating many 
5 gigabits of data, and the price of memory chips is decreasing. Processor speeds 
approaching 300 MHz are available, and speeds of several GHz are 
contemplated. 

M= To view a still or motion picture from the Internet on a personal 

0 computer, a user conventionally downloads video data from a web site by 
f0 clicking on a web link. Often, however, it is necessary to separately download 
]S (or otherwise obtain) software, e.g. Adobe Acrobat, in order to display a 
Jp& particular image format. Images are frequently compressed for transmission 
H over networks or storage on disks. Compression algorithms, such as JPEG 
^ and MPEG, using discrete cosine transfer (DCT) methods, produce serviceable 
15 images but compromise image size, image quality, definition, and acquisition 
speed. Image latency is also sacrificed. A user must wait while an entire image 
or series of images is buffered in a client side personal computer prior to 
display. Image transmission is sometimes interrupted due to network errors 
and traffic. Streaming techniques allow a user to begin viewing the images 
20 immediately while downloading, but streaming still sacrifices image quality and 
latency. 
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Currently, International Telecommunications Union Standard ITU-R 601 
for digital formats in professional video production (i.e. NTSC) requires 720 by 
486 pixels per frame in the scanned image, and an eight-bit 4:2:2 sampling of 
Y, R-Y, B-Y color components at sixty frames per second. This results in a data 
5 stream of 20 megabytes per second if the format is to remain uncompressed 
and if the images are to be viewed continuously in real time. Clearly, this is 
greater than the fastest rate for lOBaseT of one megabyte per second. A 
ju compression ratio of 5: 1 is the most that is considered desirable for production 
O marketplace image quality, but this only reduces the necessary data rate to 4 
W megabytes per second. Using 4: 1: 1 sampling, other conventional digital video 
% production techniques (e.g. DVC Pro and DV Cam) produce a marginally 
U improved data rate of 3 megabytes per second. Compression ratios of 30:1 are 
M sometimes used for previewing and editing of video images, but this only yields 

a data rate of 700 kilobytes per second. Data rates for these formats are 
15 summarized in Table 2. 



Digital Video 
Format 


Compression Ratio 


Data Rate 


ITU-R 601 


1:1 (uncompressed) 


20 megabytes/sec 


ITU-R 601 


5:1 


4 megabytes/sec 


DV (DVC Pro & DV Cam) 


5: 1 (using 4:1:1 sampling) 


3 megabytes/sec 
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ITU-R 601 



30: 1 (offline quality) 



700 kilobytes/sec 



Table 2 



Comparing this to the standard modem of 56 kilobytes per second, there 
5 is a readily apparent, significant gap between requirements for ITU-R 601 and 

present-day hardware transmission capabilities. A further compression ratio of 
y± 125:1 on an already-compressed and marginally acceptable 30:1 compressed 
O image, i.e. a total compression of 750:1, is needed to transmit ITU-R 601 data 
!■? across a 56k modem. 

W Present methods of displaying moving objects on web pages involve either 

M bit-mapped or vector approaches. Simple moving icons on a web page are 

m 

H; produced by changing only part of the image in every frame. For example, 
g Microsoft® and Netscape® browsers show moving traces around their logos 

while a processor is retrieving a page. Advertisements on web pages also 
15 display moving images. The bandwidth for these images is reduced by making 

the images smaller so that fewer bits are needed for each frame, or by slowing 

down the frame rate so that the images appear to move discontinuously. 

High definition television (HDTV) attempts to simplify the display of video 

images and reduce bandwidth by recognizing constant areas within a video 

20 .picture and retaining much of the information from a previous frame. While 
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HDTV developed concurrently with MPEG and JPEG, HDTV is broadcast- 
oriented and does not lend itself to network transmission or personal computer 
applications. 

It is expected that bandwidth will continue to be the bottleneck in 
5 network transmission for the foreseeable future. Thus, there is an outstanding 
need in the prior art to be able to send professional quality video images across 
networks through ordinary modems by taking advantage of plenary memory 
u and processor capacities within personal computers, and thereby reducing 
O reliance on transmission hardware. There is also a need to create compelling 
IP new video experiences in personal computers. 

L SUMMARY OF THE INVENTION 

M The present invention is concerned with client-side production in a 

O personal computer environment of low bandwidth images and audio. A series 

15 of still images in an image module along with a "script" module and an audio 

module are sent over a network in a client/server architecture or are read from 

a compact disk or other memory. A "director" module residing in memory (e.g. 

on hard disk) of the client personal computer uses the "script" to tell the 

computer how to execute a sequence of "moves" on the still images. These 

20 moves include, but are not limited to, cuts, dissolves, fades, wipes, focuses, 

flying image planes, and digital video effects such as push and pull. The 
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director module is either downloaded from a network on a one-time basis or 
uploaded from a floppy or compact disk. 

Production sequences are in real time, as well as being relatively smooth 
and continuous as compared to prior art network video. In order to permit 
viewing as soon as possible and to avoid caching, the script module is 
transmitted to the personal computer along with preliminary images, so 
playback begins immediately. Low bandwidth is achieved because a majority 
of the production is done at the client location and the transmission of still 
pictures, audio data and script is relatively rapid. Images are always displayed 
in real time and in full screen formats. If necessary to prevent latency delays, 
the director modules inserts stand-in from stock footage, animation and loops 
so that a viewer always has a continuous visual and audio experience. 



BRIEF DESCRIPTION OF THE DRAWINGS 

For a more complete understanding of the invention, as well as other 
features thereof, reference may be had to the following detailed description of 
the invention in conjunction with the drawings wherein: 

Figure 1 illustrates a network architecture of the present invention; 

Figures 2(a)(i)-(iii) show a first move of a bit-mapped image; 

Figures 2(b)(i)-(iv) show a second move of a bit-mapped image; 

Figures 2(c)(i)-(iv) show a third move of a bit-mapped image; 
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Figures 2(d)(i)-(iv) show a fourth move of a bit-mapped image; 
Figure 3 illustrates a flowchart for playing back visual and audio 
sequences; 

Figures 4(a) and (b) illustrate software modules for producing visual and 
audio sequences; and 

Figure 5 shows a user interface for authoring visual and audio 
sequences. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

Figure 1 shows a general overview of a client/server architecture of the 

present invention. Client personal computer 100 is connected through modem 

120 and network 125 to server 130. Computer 100 includes display screen 

140, stereo speakers 150, disk drives 160, and hard drive 170 containing a 

number of software modules. Server 130 has a memory containing other 

software modules, and server 130 is connected to a plurality of other clients 

180, 181 and also to a production client 185. Production client 185 authors 

production images 190 shown on display 140 of personal computer 100 as well 

as providing a director module for one-time plug-in at client computer 100. 

Figures 2(a) to 2(d) show selected "moves" characteristic of the low 

bandwidth television of the present invention. Figure 2(a) shows a flying video 

plane. A still picture 210 of an image to be displayed on computer screen 140 
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resides in the personal computer (Fig. 2(a) (i)). Images 230, 230' of picture 210 
are displayed on personal computer 100 such that not all of picture 210 fits 
within a display window 220, 220', but part of the picture 210 is cropped on 
the computer screen 140 as images 230, 230' (Figs. 2(a)(ii)-(iii)). Now, picture 
210 is rapidly translated within the display window 220, 220' of computer 
screen 140. Since the entire still picture 210 is already stored within the 
personal computer 100 as a bit-mapped image, there is no need to download a 
succession of images from a network or disk to give the illusion of motion. 
Moving images 230, 230' are created at the client personal computer 100 from 
the bit-mapped still picture 210 stored in the computer memory. One 
application is a pan around of a three hundred and sixty degree scene. Once 
the scene is stored as a bit-mapped still picture 210 in the computer, the 
director uses the script accompanying the still three hundred and sixty degree 
picture 210 to simulate a camera track at any position around the scene. 

Another application is a moving banner. The banner is stored as a bit- 
mapped still picture 210 (Fig. 2(a)(i)), but picture 210 is cropped such that only 
one segment of the banner is displayed at one time (Figs. 2(a)(ii)-(iii)). The 
script accompanying the banner is used by the director module to move the 
banner from left to right across screen 140, or up, down or diagonally across 
screen 140. Images 230, 230' appear to move much more rapidly and 

smoothly across screen 140 than in prior art systems because the processor of 
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computer 100 is locally working on resident picture 210 to create images 230, 
230' and a series of bit-mapped pictures 210 need not be continuously 
downloaded from a network. 

Figure 2(b) shows a focus/defocus move. A bit-mapped still picture 240 
resides in personal computer 100 along with a script module. Assume initially 
that the image is in focus and displayed on computer screen 140 as picture 
240 (Fig. 2(b) (i)). The script cues the director to activate an algorithm residing 
on the personal computer 100 which calculates and displays a series of 
defocused images 241, 242, 243 of the original bit-mapped still picture 240 
(Figs. 2(b)(ii)-(iv)). This algorithm is one of the utilities downloaded on a one- 
time basis with the director and is optimized to produce rapid calculation of the 
defocused images. Picture 240 optionally is downloaded as a pair of pictures 
240a and 240b, with picture 240a (Fig. 2(b) (i)) represented the focused or 
starting image and picture 240b (Fig. 2(b)(iv)) represented the defocused or 
target image. Preferably, the defocusing algorithm involves an optimized table 
representing iteratively defocused image states between the starting image and 
the target image. The script may indicate that the image is initially displayed as 
a defocused image and subsequently brought into focus, or vice versa. 

Figures 2(b)(i)-(iv) also can be used to illustrate a dissolve. Picture 240 is 

sent with the script module including a dissolve code. The director contains an 

algorithm for executing this dissolve, preferably using optimized tables. The 
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director thus tells computer 100 how to locally calculate dissolving images 240, 
241, 242, 243 at client computer 100, and these images 240, 241, 242, 243 are 
continuously and rapidly displayed on screen 140. 

Figures 2(c)(i)-(iv) show consecutive stages in a linear wipe. During a 

5 wipe, picture 250 is made to disappear from display screen 140 across a 
moving line 260. Line 260 may be horizontal, vertical, diagonal, straight or 
curved. As line 260 moves from one side or corner of picture 250 displayed on 

M: the computer screen 140 to the other side, picture 240 disappears from one 

0 side of the line 260 but remains on the other side (Figs. 2(c)(ii)-(iv)). Thus, 
1W picture 240 is "wiped out" or removed as line 260 moves across screen 140, 

frsis: • 

j~ and images 251, 252, 253 are calculated by computer 100 with an algorithm 

u found in the director. Wipe line 260 may also start from some place in the 

1 y 

H; center of picture 240 and process outwards, or several wipe lines 260 may 
J- 1 remove areas of picture 240 simultaneously starting from different places on 
15 picture 240. The kind of wipe that is to be executed by the director is indicated 
by the script code accompanying picture 240, and the processor of computer 
100 locally calculates the wipe with algorithms found in the director. As the 
wipe line traverses across the image, a screen optionally is revealed to have a 
particular background 270 (Fig. 2(c)(iv)). Background 270 may be another bit- 
20 mapped image such as a photograph, text or graphic, or the background may 
be a solid. 
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Figure 2(d) shows a biaxial wipe. Pieces 271, 272, 273, 274 of a bit- 
mapped picture 240 appear to split apart, and each piece 271, 272, 273, 274 
moves in a different direction towards the edges of the display screen 140 (Fig. 
2(d)(i)-(iv)). One example splits picture 240 along horizontal and vertical axes 
280, 281, with each piece 271, 272, 273, 274 moving towards a respective 
corner of the display screen 140. There also may be two pieces moving along a 
diagonal line towards opposite diagonal sides of the screen. Once the wipe 
begins, the absence of the image optionally reveals another bit-mapped image, 
a blank screen, a particular color or texture. 

Low bandwidth television produces a sequence of moves on still bit- 
mapped images specified by an accompanying script. The production sequence 
can be rapidly and consecutively strobed and repeated in a particular order, or 
the sequence can be strobed and repeated in a different order. Repetition and 
looping of sequences implies that any production sequence has an arbitrarily 
long and potentially infinite duration. A production sequence may consist of 
combinations of still images, high resolution photographs, text graphics, high 
resolution text, and animated computer graphics. While the present 
embodiment contemplates that the director primarily operates on still images, 
short video clips residing as stock footage with the director module may 
optionally be utilized. 
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Low bandwidth television assumes full screen and real time display of 
images. In contrast to prior art systems where image size must be scaled and 
quality reduced to conserve bandwidth, the present invention improves the 
viewing experience by requiring that images cover the entire screen of the 
personal computer. Larger size and higher resolution pictures Eire possible 
because a majority of the production work is contemporaneously performed by 
the director at the client location rather than prior to network transmission. 
Real time display is achieved because the image and audio modules are 
transmitted quickly across the network due to their small bandwidth. The 
director further guarantees a real time experience by inserting stock footage, 
looping and stretching whenever image data is delayed due to network latency. 

Each image module is generally synchronized with an audio track that is 
sent with the script. The audio track optionally includes music tracks, Foley 
effects, and voiceovers. An audio engine has a capability of mixing multiple 
audio tracks and adding special audio effects such as reverb and audio delays 
in real-time. The director module includes a high quality audio synthesizer 
having a file size of about 20 megabytes. 

One major difference between the low bandwidth production system of 
the present invention and prior art video production systems is the degree to 
which a finished product is sent over a network or stored on a disk. Prior art 

Internet video devices (e.g. MPEG) send a finished product over the network, 
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while the present invention sends only a partial product and a script and then 
finishes the video production at the client station with the director. Much 
greater bandwidth is required for the prior systems of sending a finished series 
of images over the network than it does to send a partially completed set of 
images with a script describing how the images are to be animated, and then 
finishing the animation of the images at the client computer. In a disk storage 
environment, much more disk space is required to store all of the pixels of a 
series of images than to store one image and script code representing how the 
images are to be animated. 

Furthermore, the video production method of the present invention is 
much faster than prior art methods despite the reassembly time for still image 
production at the client. The speed of the prior art method of downloading 
video images from the Internet is limited by a bottleneck at the modem. By 
contrast, while the video production of the present invention is uncompleted at 
the time it arrives at the client computer, the processor reconstructs the 
production from the images and the script much more quickly than the delay 
occasioned at the modem. 

LBTV has a number of advantageous characteristics. It uses the same 

audio and visual language of film and video production standards. Smooth 

and continuous motion is produced in real-time as compared to standard 

methods of viewing images from networks. There is no image latency because 
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the image stills and script are transmitted rapidly in comparatively small files. 
Moving images are displayed in real-time because the director quickly 
calculates the production sequence at the client computer from the stills and 
script. The images are displayed at sixty fields per second (in NTSC) with anti- 
aliased graphics, high-resolution imagery, full-screen displays and high-quality 
audio. These capabilities are realized because the majority of the work is done 
by exploiting the processor and memory at the client computer. 

Although digitized video clips may be used with LBTV, their large 
bandwidth implies that they are utilized sparingly. However, clip bandwidth 
can be decreased with keys to reduce their size, or with other special effects 
such as strobing or posterization. In strobing, every fifth video frame is 
displayed and frozen. Stock footage stored at the client computer may also be 
used since it requires no network transmission time. 

The present invention also provides stand-in and loops to permit an 

immediate and continuous viewing experience without caching. Initially, only 

the script module and the earliest part of the image data from the image 

module are sent across the network. Thus, presentation of images and sound 

begins immediately for the viewer without downloading of the entire image file. 

Neither is it necessary for the image and sound data to cache in the client 

computer memory. To prevent latency problems, the director inserts stock 

footage as stand-ins or causes the images already received to loop or stretch in 
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the production sequence. Therefore, in contrast to prior art systems where the 
visual stream is interrupted or the viewer must wait while the images are 
downloaded, the director ensures a continuous viewing experience. 

The present invention is also applicable to receiving a production module 
comprising a script module, an image module, and an audio module, from a 
disk drive, e.g. a CD-ROM, rather than obtaining this module over a network. 
While digital video disks (DVDs) provide for real time viewing at approximately 
sixty frames per second, low bandwidth production techniques further increase 
the number and run time of programs that can be stored on a single DVD. 
Moreover, LBTV does this without data compression. 

Figure 3 is a flowchart of the steps in creating and viewing a low 
bandwidth television production. A user first loads a plug-in in step 310. The 
plug-in contains a director module including algorithms to perform all of the 
video "moves," as well as stock footage and other utilities. The loading is done 
on a one-time basis, although the plug-in may be updated periodically, e.g. 
when new algorithms are added to the director (step 315). The plug-in may be 
downloaded from a network server, or the plug-in may be uploaded from a disk 
drive, for example, a magnetic disk or a CD-ROM. 

A particular video production begins in step 320. The production module 

includes an image module, an audio module and a script module. Initially, 

only the script module and first viewing parts of the image and audio modules 
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are transmitted over the network so that viewing begins immediately without 
caching. Viewing is initiated either by clicking on a link in a web site and 
receiving transmitted data from a network (e.g. the Internet) via a server, or by 
reading from a disk drive, for example, a magnetic disk or a CD-ROM. 

The director module uses the script module to generate initial video and 
audio sequences from the image module and the audio module (step 330). The 
video and audio sequences are played on the video screen and through stereo 
speakers of the personal computer (step 340). Meanwhile, more data from the 
image module and audio module are loaded across the network into the client 
computer (step 370). The director module continues to work on the newly 
received data from the image and audio modules with cues from the script 
module to generate new visual and audio sequences. 

If there is a gap at any time in the production due to latency or data 
transfer problems (step 345), the director maintains a continuous real time 
presentation by inserting stock footage or providing looping (step 360). As long 
as there is more data being received from the network (step 365), the director 
continues to load data from the image and audio modules (step 370). When 
program data transmission is complete, a user may return to play another 
video and audio sequence (step 380), or terminate the program (step 390). 

Figures 4(a) and (b) illustrate software components of the various 

modules of the present invention. The plug-in comprises director module 410, 
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which includes full screen transition algorithms 420 and partial screen effects 
algorithms 430 (Fig. 4(a)). Examples of full screen transitions are moving 
bitmaps 421, wipes 422, and animation programs 423. Director 410 further 
comprises instrument sample libraries 440, stills 450, and stock footage 460. 
5 Stock footage 460 may include short video clips. Each of the modules is on the 
order of 20 megabytes. Software algorithms of director module 410 are 
designated by Global Unique Identifiers (GUIs) to permit indexing of algorithms. 
M. Production module 470 includes script module 475 with commands in 

O an edit decision list (EDL), image module 480 having bit-mapped images of the 
lfP still pictures utilized in the production, including photographs 481, graphic 
]£ images 482, and short video clips 483 (Fig. 4(b)). Audio module 490 includes 
M, music score 491, voiceovers 492 and sound effects 493. Production module 
H 470 further includes optional new effects 495, which may be identified as 
jf-f needed for a specific production module 470. New effects software algorithms 
15 495 are identified by the edit decision list of script module 480 via respective 
GUIs to determine whether algorithms 495 are already present in director 
module 410. 

Figure 5 illustrates a User Interface (UI) for an authoring tool to create 

production modules 470 as a series of pages 510. A user first specifies page 

20 511 corresponding to a first in a series of effects in a sequence. Button bars 

520, 521, 522, 523 are selected to edit pages 510. EDL commands are 
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attached to page 51 1 by clicking on button bars 520-523 to create script 
module 480. Bit-mapped still image files are attached to page 511 to form 
image module 490, and sound effects files are attached to page 51 1 to make 
audio module 480. Preview window 530 displays still bitmapped images, and a 
list of file names for image, audio and script (EDL) files is displayed in preview 
window 540. Once a bitmapped image, audio file, and edit decision list are 
associated with page 511, the process is repeated for pages 512, 513, 514, etc. 
Production module 470 is complete once all of pages 510 are created and 
assembled. A production editor can then go back and re-edit any of the pages. 

A completed production module 470 is stored on server 130 as a disk 
file(s), and client computer 100 contacts server 130 when a viewer desires to 
watch the program. Client-side production may begin immediately upon 
sending the EDL with some initial program content, and client computer 100 
performs production on-the-fly. Additional content not yet present at client 
100 is transmitted in temporal order. Even if a given section of the program 
occurs before all of the required content has been transferred, the playback 
engine ensures continuity using stand-ins such as line drawings and looped 
sequences. 

Although the present invention has been described in its preferred 

embodiments, those skilled in the art will appreciate that alternate 

embodiments, not specifically described herein, may be deduced without 
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departing from the spirit and scope of the invention, which is limited only by 
the following claims. 
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